In view of the problems of the current Convolutional Neural Network （CNN） using end layer features to recognize facial expression， such as complex model structure， too many parameters and unsatisfactory recognition， an optimization algorithm based on the combination of improved CNN and Support Vector Machine （SVM） was proposed. First， the network model was designed by the idea of continuous convolution to obtain more nonlinear activations. Then， the adaptive Global Average Pooling （GAP） layer was used to replace the fully connected layer in traditional CNN to reduce the network parameters. Finally， in order to improve generalization ability of the model， SVM classifier instead of the traditional Softmax function was used to realize expression recognition. Experimental results show that the proposed algorithm achieves 73.4% and 98.06% recognition accuracy on Fer2013 and CK+ datasets， which is 2.2 percentage points higher than the traditional LeNet-5 algorithm on Fer2013 dataset. Moreover， this network model has simple structure， less parameters and good robustness.